Repository navigation
feat(rules): semantic rules with closed verdicts, Jev and Anthropic judges - #151
Conversation
Code reviewNo issues found. Checked for bugs and CLAUDE.md compliance. |
mindsocket
left a comment
There was a problem hiding this comment.
The docs leave a bit to be desired in terms of content design and approachability. Keep focused, move more toward ASD-STE100, refer to other docs/sections, (it's ok to repeat small tidbits when necessary and non-obvious), consider information architecture (eg rules doc should capture key ideas, and semantic rules should point to them and take as read instead of randomly tossing things in), introduce concepts once and properly before sections that need them described.
| @@ -0,0 +1,103 @@ | |||
| # Semantic Rules | |||
|
|
|||
| A semantic rule is a rule a model judges instead of a JSONata expression: "is this opportunity framed in the customer's voice?" rather than "does this field exist?". Semantic rules are **opt-in**. Nothing is sent to a model unless you pass `--semantic` (or enable it in config), and a space with no semantic rules is unaffected. | |||
There was a problem hiding this comment.
This is too verbose. No need to mention jsonata, "rather than", 2 long sentences for how to run it when 1 short statement will do. Apply same thinking throughout updated docs.
There was a problem hiding this comment.
Rewritten in 6921788. The intro is now two short sentences: what a semantic rule is and when to use one. The rest of the doc gets the same treatment.
| } | ||
| ``` | ||
|
|
||
| 2. **Choose a judge** in `config.json`. The judge `type` is always explicit; keys in your environment are never picked up unless you name the type: |
There was a problem hiding this comment.
This is confusing language. Is this a quickstart or not? Just say "use built in anthropic type and it will pick up ANTHROPIC_API_KEY" or something
There was a problem hiding this comment.
The quick start now says: set OPENROUTER_API_KEY, add { "judge": { "type": "jev" } }. One sentence gives Claude as the alternative ("type": "anthropic" with ANTHROPIC_API_KEY).
|
|
||
| with `ANTHROPIC_API_KEY` set. See [Judge configuration](config.md#judge-semantic-rules) for every field. | ||
|
|
||
| 3. **Run it**: `sctx validate <space> --semantic`. To turn it on by default, set `semantic: true` (or `judge.autoRun`) in config; `--no-semantic` overrides that for a single run. |
There was a problem hiding this comment.
Again, not quickstart material.
Aside, why are there 2 ways to turn it on (there could be a good reason), and where do these things go. These random bits of assumed knowledge suggest poorly thought through content design.
There was a problem hiding this comment.
There was no good reason for two switches. judge.autoRun duplicated semantic, so I removed it from the code and docs. The quick start just runs with --semantic. A new section, "Turning semantic rules on", covers --semantic, "semantic": true (top level or per space, linked to config.md) and --no-semantic.
|
|
||
| ### Verdicts, severity and guidance | ||
|
|
||
| The judge must choose exactly one verdict. A closed set keeps results comparable between runs and maps directly onto decision models, which return a choice with confidence and probabilities rather than prose. |
There was a problem hiding this comment.
Cut. The verdicts section now says the judge gives one verdict per rule and node, from a fixed set, and that a fixed set makes results comparable between runs.
|
|
||
| The judge must choose exactly one verdict. A closed set keeps results comparable between runs and maps directly onto decision models, which return a choice with confidence and probabilities rather than prose. | ||
|
|
||
| - **A verdict is a classification, severity is the consequence.** `severity` as a map decides what each verdict means for the user: `error`, `warning` or `info`. A verdict left out of the map passes, and `ignore` states that explicitly: the rule ran and the verdict was recorded, but nothing is reported. At least one verdict must pass. |
There was a problem hiding this comment.
"passes" isn't well defined, assumes that "passing" a rule is a thing, and I'm not sure that language is uniquitous and clear.
There was a problem hiding this comment.
Agreed. rules.md now defines a violation under Severity: a rule either reports a violation (with a severity) or reports nothing. The semantic docs say a verdict "reports a violation" or "reports no violation", and never "passes".
| "when": "current.status = 'ready' and $toMillis(current.due) - $millis() < 604800000" } | ||
| ``` | ||
|
|
||
| `when` works on expression rules too. It saves money on semantic rules because the cheap test runs first. |
There was a problem hiding this comment.
blah blah.
Better to document "when" once in rules, and how it helps optimise rule running and semantic rule costs. Then this doc needs little more.
There was a problem hiding this comment.
Done. rules.md documents when once, including that it costs almost nothing and saves a model call on each node it excludes from a semantic rule. semantic-rules.md only links to it. --rule-filter also moved to rules.md, because it applies to all rules.
| | `openai` | Any OpenAI-compatible chat endpoint: OpenAI, Ollama, vLLM, Groq, OpenRouter chat models | Set `endpoint` for non-OpenAI servers; the key is optional for local ones | | ||
| | `jev` | TypeSafe's Jev decision model | Closed-set answers with calibrated probabilities, one request per node. Uses an OpenRouter key | | ||
|
|
||
| Prefer a decision model when you can: results are repeatable, cheap, and come with a confidence. Use a chat LLM when the rubric needs open reasoning or evidence text. |
There was a problem hiding this comment.
The preferred model sure is buried here. Put it in the quickstart.
There was a problem hiding this comment.
Jev is now step 2 of the quick start, as the recommended judge. Choosing a judge lists it first. In f6b3e04 I verified Jev against the live OpenRouter endpoint, and bun run test:judge now covers it.
| |---|---|---| | ||
| | `anthropic` | Claude models | Default model `claude-haiku-5-5` | | ||
| | `openai` | Any OpenAI-compatible chat endpoint: OpenAI, Ollama, vLLM, Groq, OpenRouter chat models | Set `endpoint` for non-OpenAI servers; the key is optional for local ones | | ||
| | `jev` | TypeSafe's Jev decision model | Closed-set answers with calibrated probabilities, one request per node. Uses an OpenRouter key | |
There was a problem hiding this comment.
To check: are error messages helpful when someone's set jev but not provided an OPENROUTER_API_KEY
There was a problem hiding this comment.
Checked from the CLI with OPENROUTER_API_KEY unset. It exits 1 with: Error: Judge "default" (jev) has no API key. Set OPENROUTER_API_KEY in the environment, or "apiKey" / "apiKeyEnv" on the provider.
| - **`references/schema-design.md`** — process for designing a schema from existing content | ||
|
|
||
| For CLI and config reference, use `sctx docs <topic>` (topics: `concepts`, `config`, `schema`, `rules`). | ||
| For CLI and config reference, use `sctx docs <topic>` (topics: `concepts`, `config`, `schema`, `rules`, `semantic`). |
There was a problem hiding this comment.
rename doc to semantic-rules so this makes more sense
There was a problem hiding this comment.
Renamed: sctx docs semantic-rules. I updated SKILL.md, the README, rules.md, schema-authoring.md and the docs command help to match.
| // OpenAI-compatible endpoints (Ollama, vLLM, ...) may need no key. | ||
| if (!apiKey && type !== 'openai') { | ||
| throw new JudgeConfigError( | ||
| `Judge "${name}" (${type}) has no API key. Set ${keyEnv} in the environment, or "apiKey" / "apiKeyEnv" on the provider.`, |
There was a problem hiding this comment.
Looks like this is the friendly error message. 👍
|
Re: the review on docs content design (6921788):
Also in this push (f6b3e04):
|
| | `apiKeyEnv` | The environment variable that holds the key. Defaults: `OPENROUTER_API_KEY` (jev), `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`. A local `openai` endpoint can need no key. | | ||
| | `apiKey` | The key itself. Use `apiKeyEnv` instead, so that keys stay out of config files. | | ||
|
|
||
| sctx creates a provider only when it uses it, so a provider that is not used needs no key. The environment holds only keys: no environment variable selects or configures a provider. |
There was a problem hiding this comment.
Still overexplaining. Also, don't use sctx shorthand in documentation. sctx is the cli command and informal shorthand, structured-context is the proper project name.
Suggested: "The validation engine creates a provider on when it uses it."
There was a problem hiding this comment.
Done in 5c02201: the section is now "Model providers and judge", merged per the next comment, and prose in docs uses structured-context (the Documentation Principles section in AGENTS.md now records this).
| - An **expression rule** (the default) is a JSONata `check` that must return `true`. | ||
| - A **semantic rule** asks a model to judge a node. See [Semantic rules](semantic-rules.md). | ||
|
|
||
| This page applies to both kinds, unless it says otherwise. |
| | `id` | yes | Unique rule identifier | | ||
| | `category` | yes | `validation` \| `coherence` \| `workflow` \| `best-practice` | | ||
| | `severity` | no | `error` \| `warning` \| `info`; overrides the category default (see [Severity](#severity)) | | ||
| | `severity` | no | `error` \| `warning` \| `info`; overrides the category default (see [Severity](#severity)). A semantic rule can map verdicts to severities | |
There was a problem hiding this comment.
If this is covered in severity section and/or semantic-rules, don't add random snippets about semantic rules here. This applies generally as a principle too, unless the callout is of critical importance in context.
| | `anthropic` | Claude. Default model: `claude-haiku-5-5` | `ANTHROPIC_API_KEY` | | ||
| | `openai` | Any OpenAI-compatible chat endpoint, for example OpenAI, Ollama or vLLM. Set `endpoint` for servers other than OpenAI. Local servers do not need a key | `OPENAI_API_KEY` | | ||
|
|
||
| sctx uses only the provider that you configure. It does not choose a provider from the keys in your environment. See [Model providers](config.md#model-providers) and [Judge](config.md#judge-semantic-rules) for all fields. |
There was a problem hiding this comment.
s/configure/specify/ might be better. You can specify "jev" without configuring it (with a key) per-se.
| @@ -0,0 +1,5 @@ | |||
| --- | |||
| type: opportunity | |||
There was a problem hiding this comment.
Consider converting this whole space fixture into a single json or jsonl of spacenodes, or whatever the judge test needs, unless the test is necessarily to exercise more of the parse/validate pipeline.
| }; | ||
|
|
||
| function validate(...args: string[]): { exitCode: number; output: Output } { | ||
| const result = Bun.spawnSync(['bun', 'run', 'src/index.ts', 'validate', 'judge-test', '--json', ...args], { |
There was a problem hiding this comment.
Do we need to do quite the extensive dance with a process call and full parse just because of environment? Could this test be simplified?
| ], | ||
| "schema": "/path/to/custom/schemas/my-general-schema.json" | ||
| "schema": "/path/to/custom/schemas/my-general-schema.json", | ||
| "modelProviders": { |
There was a problem hiding this comment.
We discussed not having default config accidentally enabling model calls, but perhaps requiring --semantic is already enough.
There was a problem hiding this comment.
Agreed with the musing: kept explicit modelProviders, but config.example.json now uses the shorthand ("judge": { "type": "jev" }, 5c02201), so the example no longer implies that default config enables model calls — a run still has to ask with --semantic/--judge.
Semantic rules (mode: semantic) declare a closed verdict set (verdicts, failOn) and are judged per node, all of a node's rules in one provider call. Judges: Anthropic (default claude-haiku-5-5), OpenAI-compatible, Jev via OpenRouter decisions API, fake. Semantic rules run inside cascade gating (#115) after deterministic rules and only with --semantic. --rule-filter narrows rules by mode, category, severity, type or id.
…es, accept jev in config Addresses review on the semantic rules PR.
Per the definition of done: config.md (judge/semantic keys), concepts.md (semantic rule, judge, verdict; gating no longer 'future'), api.md (judging section, option lists), AGENTS.md (src/judge, rule-filter), config example, and the remaining src/api.ts exports. Adds a config test for provider jev.
…e/provider, split semantic docs, trim test fixtures
…e, hooks never judge
…ondition, named judge providers and --judge
…expression errors as violations - judge config uses `type` in the flat and named forms; `use` becomes `default` - remove SCTX_JUDGE_* overrides (config, flags and apiKeyEnv cover them); document where a setting lives - a check or when that cannot be evaluated is an error violation, not a silent pass - okf: stale_after ordering is a validation rule; drop redundant explicit severities
…Run; rename docs topic to semantic-rules rules.md now owns the shared ideas (violation, when, rule filtering) and semantic-rules.md covers only what differs, in shorter sentences. Quick start leads with Jev. judge.autoRun duplicated the semantic config key, so semantic is now the only config switch. sctx docs semantic becomes sctx docs semantic-rules.
… and gatedRules missing from JSON bun run test:judge sets SCTX_LIVE_JUDGE_TESTS=1 and runs judge-test/ against a fixture space with each provider whose API key is set. Without the variable the tests skip, so bun test stays offline. The validate action did not pass --judge through, so naming a judge there ran no semantic rules. validate --json also left out gatedRules, which the docs list.
… API; guard API surface in AGENTS.md No consumer needs the judge providers, factory, judge types or rule filter, so src/api.ts and docs/api.md return to their previous scope. AGENTS.md now requires explicit agreement before any export is added.
…types Violations and coverage gates already report what users act on, so the per-evaluation and per-gated-rule records go, with their JSON fields and CLI summary lines. The judge input node is a Pick of SpaceNode, the rule spec no longer carries failOn (severity stays out of the judge's input), and JudgeConfig extends the provider definition with a real type union.
… for semantic rules Config now separates how to reach a model (top-level modelProviders) from which one judges (judge.provider, top level, with a space override). Config no longer turns semantic rules on: --semantic or --judge <provider> (or the API options) ask on each run, so --no-semantic and the hooks' pin go. Rules no longer name a judge: that is config, not schema. The judge is built once per run, and only when a semantic rule is to be judged.
… to judge validateSpace and validateFile take an optional resolved judge instead of semantic, a judge name and an injected provider: a judge means judge. --semantic and --judge are CLI concerns, turned into a judge by judgeForRun. validateRules follows the same shape internally.
A judge names either a bundled provider type ({ "type": "jev" }) or an
entry in modelProviders ({ "provider": "..." }); a modelProviders entry
wins over a bundled type of the same name. No judge in config means no
model runs, and a run still asks with --semantic or --judge.
Docs: merge config.md's provider and judge sections, adopt a
Documentation Principles section in AGENTS.md (one idea lives in one
place; document what it does; structured-context in prose; front-load
the happy path), and trim rules.md per review.
judge.provider names a provider: a modelProviders entry first, else a bundled provider (jev, anthropic, openai). A user entry replaces a bundled one of the same name, keeping type defaults for fields it leaves out. The fake provider stays test-only. The judge-test fixture space is gone: the live provider test now judges one inline node directly (SCTX_LIVE_JUDGE_TESTS=1, per-provider key check), and pipeline coverage stays with the existing fake-judge tests.
Provider definitions name a wire API (provider.api: openai-completions, anthropic-messages, openrouter-decisions, typesafe-systemone) instead of a model-flavoured type. The two decisions APIs share one implementation. Hydration happens in the factory, not the providers: explicit value > named env var (set-but-empty is an error) > API default. Provider implementations read no environment; only the key has a per-API default variable. The bundled catalogue now carries vendor bundles: openrouter-jev (typesafe/jev-latest), typesafe-jev, anthropic, anthropic-haiku-5-5, anthropic-env, openai, openai-env, openrouter. New sctx providers lists bundled and configured providers with their effective config, keys never shown.
…ctx providers The openrouter bundle spoke openai-completions with no endpoint, so its key would be sent to api.openai.com. It now pins https://openrouter.ai/api/v1. sctx providers shows each provider's effective endpoint, and bundle defaults are pinned by tests.
…setup - new openai-decisions API and OpenAIDecisionsJudgeProvider (one request per node, choice questions from the rule's verdict set; verified live against api.openai.com/v1/decisions) - new bundles: openai-decisions, openrouter-openai-decisions; openrouter chat bundle renamed to openrouter-openai - model constants live in the provider modules; the factory imports them (recommended models per vendor are this project's judgement; pinned bundles keep their literal) - OpenRouter model slug verified: ~typesafe/jev-latest resolves on the decisions endpoint; typesafe/jev-latest does not - docs: bundled-provider tables replaced by "sctx providers" plus the wire-API table and naming conventions - bun run test:models smoke-checks each bundled model resolves (keyless where the endpoint allows; skip per provider without a key) Co-authored-by: Claude <noreply@anthropic.com>
d8ceb90 to
d997be9
Compare
Implements the core of #114. Left open for the remaining questions there: caching, retry and concurrency, and the evals layer (#116).
Builds on cascade gating (#148, merged). Supersedes #149, which GitHub auto-closed when its stacked base branch was deleted.
What
Semantic rules (
mode: semantic) judged by an LLM or a decision model, running inside cascade gating. Opt-in per run and inert otherwise: nothing is sent anywhere unless a run passes--semanticor--judge <provider>(or an API caller passes a judge). Config cannot turn judging on. A schema with no semantic rules behaves exactly as before.Closed verdicts. A rule declares
verdicts; defaultpass/fail. A verdict is a description for the judge, or{description, guidance}whereguidanceis shown to the user when it fires (never sent to the judge). Judges return a verdict plus optional confidence, probabilities and evidence.Verdict is classification, severity is consequence.
severityis a level, or for semantic rules a map{verdict: error|warning|info|ignore}. Verdicts left out of the map pass;ignoremeans the rule ran and the verdict is recorded but not reported. Required when custom verdicts are declared (there is nofailOn). Gating follows the verdict's severity.when. An optional JSONata applicability test on any per-node rule, evaluated before the rule (so before any judge call). Where it is nottruethe rule is skipped silently, neither passed nor gated.Model providers, then the judge. Config separates how to reach a model from which one judges. Top-level
modelProvidersnames each provider (type,model,endpoint,apiKeyEnv).judge.providerpicks the judge, at the top level or overridden per space (for example, a private space on a local model). The provider is never inferred from API keys in the environment, so a key set for another tool is never spent unless the user chose that provider. There are no env overrides: config says what, flags override per run, env holds secrets only. A misconfigured judge is a plain CLI error, raised before any request and only when a semantic rule is to be judged.Rules don't choose a judge. That is config, not schema, so schemas stay portable across configs.
--judge <provider>picks a different configured provider for one run (implies--semantic). Config-level overrides for classes of rules are a possible follow-up.Providers (
src/judge/provider/):anthropic(defaultclaude-haiku-5-5),openai(any OpenAI-compatible chat endpoint, key optional for local servers),jev(TypeSafe decision model through OpenRouter's alpha decisions API,OPENROUTER_API_KEY),fakefor tests. All are plainfetchcalls; no SDK dependencies.Per-node batching. All of a node's semantic rules go to the judge in one call. Chat providers make one request per rule inside it; Jev answers them in one request.
Gating. Expression rules run first, then the judge. Semantic rules are skipped on gated nodes and gate their subtree per
gatesDescendants. A judge error is reported as a violation but never gates.validate-fileis cheap. Gating is computed over the whole space, but the judge is called only for the file and for its ancestors' gate-capable semantic rules (top-down, stopping at the first gate): at most one call per ancestor level plus one.--rule-filter dim=values(mode, category, severity, type, id; repeatable; a verdict map matches any severity it can report) and--judge <provider>onvalidateandvalidate-file.Rule contract.
checkis required unlessmode: semantic; semantic rules cannot declarecheckorscope: global;verdicts,rubric,judgeand a severity map are semantic-only;whencannot combine withscope: global.sctx schemas showlists a rule's mode, severities, judge andwhen.Docs.
docs/rules.md(now "Rules") owns the shared ideas: both kinds of rule, violations,whenand--rule-filter. Newdocs/semantic-rules.md(sctx docs semantic-rules) covers only what differs, starting with a quick start. Judge fields are indocs/config.mdand terms indocs/concepts.md.No new public API.
structured-context/apiexports are unchanged.validateSpaceandvalidateFilegain optionaljudge(a resolved judge: present means judge) andruleFilteroptions, andRuleViolationgains optionalevidence,verdictandguidance. Judge providers, the factory and the judge types stay internal: building a judge is a core and CLI concern, and--semantic/--judge <provider>are resolved into a judge at the CLI.Not in this PR
guidancecovers coaching for now.includeGatedis still one switch for both rule kinds (it gates deterministic and semantic rules alike). Splitting them, so an editor can un-gate cheap rules while judged ones stay gated, is a follow-up.POST /v1/decisions,inputplusquestions) with a different wire shape from OpenRouter/TypeSafe's (stateplus keyedquestions); both are candidates for further providers behind the same interface.Testing
660 unit tests (
bun run test); typecheck and biome are clean. New tests cover verdict resolution, prompt/response parsing, each provider (HTTP mocked), provider and judge config (including that no provider is ever inferred from the environment, and the space and run overrides), rule contract, rule filter, batching, gating interaction, which nodesvalidate-filejudges, verdict severity/ignore/guidance and per-verdict gating,when, and the CLI--judgewiring.bun run test:judgesetsSCTX_LIVE_JUDGE_TESTS=1and runsjudge-test/against a fixture space with each provider whose API key is set (OPENROUTER_API_KEY,ANTHROPIC_API_KEY,OPENAI_API_KEY). Without the variable they skip, sobun teststays offline. Jev and OpenAI pass live; Anthropic has not run yet, because no key was available. The OpenRouter decisions endpoint is still alpha.checkorwhenthat cannot be evaluated (e.g. a syntax error) is reported as anerrorviolation (Rule error: ...) instead of silently passing; it does not gate descendants.